Traditionally, humans have been the primary source of Internet traffic experienced by network services (e.g., Web servers). However, more recently, a large increase has occurred in the number of requests to network services that are initiated by automated systems and certain computer programs, which are referred to as automated agents, automated robots or so-called “bots,” provide tools that automate tasks, such as crawling through webpages to scan and gather keywords, text and links. Web Bots may exhibit sophisticated functionality and place widely varying demands upon Web servers to which the HTTP requests are directed. In some instances, the Web Bots may seek to access a Web server for legitimate reasons or malicious reasons. Regardless of the intent of the Web Bot, the demands imposed by Web Bots at times may overburden the capability of a Web server. When a Web server experiences excessive traffic, among other things, the Web server may begin to respond more slowly to human-based traffic or experience more significant consequences. Website designers and operators such as online merchants are engaged in an ongoing effort to manage Web Bot-based traffic seeking access to Web servers.
Bot detection systems have been proposed that seek to prevent the Web Bot activity from degrading the performance of the Web servers in connection with human-based requests/traffic. Conventional Bot detection systems typically identify a HTTP request from Bots based on the IP (Internet Protocol) address associate with the source of the HTTP request. The Bot detection systems scan incoming HTTP requests for IP addresses known to be associated with Bot systems. When traffic from a Bot associated IP address is identified, the traffic is handled in a particular manner. For example, the traffic may be quarantined when the IP address is associate with a Bot known to have a malicious intent, known to hack websites, known to seek individual user information or otherwise. As another example, when the IP address is associated with Bot activity (or Bot originated) that is considered “safe”, the incoming request may still be managed in a manner that prevents degradation of human-based traffic.
However, some conventional Bot detection systems are not sufficiently robust and may inadvertently block traffic from customers (e.g., human based requests/traffic). In addition, Bot detection systems may return numerous “false positives” indicating apparent Bot attempts to access a website. In many instances, traffic identified as “Bot originated” may not in fact be Bot originated. For example, a single IP address may be utilized in connection with numerous users that are not related with one another, namely users associated with human originated traffic and users associated with Bot originated traffic. Thus, when the IP address is used as the basis to identify Bots, human traffic may be inadvertently identified as Bot traffic.
Accordingly, it is desirable for improved methods, systems and computer program products that detect and differentiate between human and Bot traffic.